(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(1 9) World Intellectual Property Organization 

International Bureau 




ill ii i iii iiiiiii ii ii iii mini inn mil mi mi 



(43) International Publication Date (10) International Publication Number 

28 February 2002 (28.02.2002) PCT WO 02/17633 A2 



(51) International Patent Classification 7 : H04N 7/16 (74) Agent: GROENENDAAL, Antonius, W., M.; Interna- 

tionaal Octrooibureau B.V., Prof. Holstlaan 6, NL-5656 

(21) International Application Number: PCT/EP0 1/09634 AA Eintlhoven (NL)- 

(22) International Filing Date: 13 August 2001 (13.08.2001) < 81 > Designated States (national): CN, JP, KR. 

(25) Filing Language: English ( 84 > Designated States (regional): European patent (AT, BE, 

CH, CY. DE, DK, ES, FT, FR, GB, GR, IE, IT, LU, MC, 



(26) Publication Language: English 



NL, PT, SE, TR). 



nm p • • Published: 

{ > 1 1 y • — without international search report and to be republished 



09/643,186 21 August 2000 (21.08.2000) US 



upon receipt of that report 



(71) Applicant: KONINKLIJKE PHILIPS ELECTRON- For hv0 _ letter codes and other abbreviatl0ns> refer to the "Guid- 
ICS N.V. [NL/NL]; Groenewoudseweg 1, NL-5621 BA 
Hindhovcn (NL). 



:e Notes on Codes and Abbreviations " appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



j (72) Inventors: DIMITROVA, Nevenka; Prof. Holstlaan 6, 
j NL-5656 AA Eindhoven (NL). DEVARA, Kavitha, V.; 
j Prof. Holstlaan 6, NL-5656 AA Eindhoven (NL). 



(T) (54) Title: METHOD AND SYSTEM FOR ACTIVE MODIFICATION OF VIDEO CONTENT RESPONSTVELY TO 
PROCESSES AND DATA EMBEDDED IN A VIDEO STREAM 

(57) Abstract: Video, or other media data, are synchronized with procedure data streams that modify the video stream. When 

^ the video is played, reproduced, or rebroadcast, the video is modified by the procedures defined in the procedure data stream. In 

a stateless embodiment, the procedure stream includes commands that are executed immediately upon receipt by an interpreter. 

In a particular embodiment, the procedure data stream is incorporated directly in the media data stream and separated out by a 
^ demultiplexer. In a still more particular embodiment, the combined media/procedure data stream may be fractured into portions 
^ while still carrying code appropriate to the portion removed from the whole. 



WO 02/17633 



PCT/EP01/09634 



Method and system for active modification of video content responsively to processes and 
data embedded in a video stream 



BACKGROUND OF THE INVENTION 

Field of the Invention 

The invention relates to video systems in which video content is actively 
5 modified according to conditions at the delivery site, preferences of a user, or other 
conditions and more particularly to such systems where the procedures controlling 
modification are embedded in or otherwise synchronized with the video stream. 

Background 

10 Certain portions of video content may be considered to be objectionable and 

other portions unobjectionable. This has inspired some to propose that markers be 
incorporated in video data to indicate the portions that might be considered objectionable for 
some audiences. An application running at the delivery location recognizes markers in the 
video stream and selectively mutes or deletes passages responsively to them. For example, 

15 PCT Application WO 98/21891 published May 22, 1998 for INHIBITION OF A TV 

PROGRAMME DISPLAY ACCORDING TO THE CONTENTS, describes blanking out 
delimited portions of a video stream marked as containing objectionable subject matter. 

Application GB 2 284 914 filed December 18, 1993 for DISCRETIONARY 
VIEWING CONTROL describes a system that places restrictions on television programming 

20 based on time of day, the identity of the program, the rating of the program, etc. based on 
conditions defined at the delivery point. Again, the result is to delete or inhibit the video 
signal when certain conditions are present. 

Application WO 83/02208 published June 12, 1983 for METHOD AND 
APPARATUS FOR EDITING THE OUTPUT OF A TELEVISION SET, describes a system 

25 that filters content based on markers in the video and an application running on a television 
set top box. The markers inserted in the signal are graded according to the content. 
UK Application GB 2315175 filed July 10, 1997 for RESTRICTING ACCESS TO VIDEO 
MATERIAL describes a system in which video content is allowed to be transmitted or 
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reproduced based on a characterization of its content. The filtering application is predefined 
at the nexus between transmission and recording or viewing. 

US Patent No. 5,751,335 filed Feb. 23, 1996 for VIEWING RESTRICTING 
METHOD AND VIEWING RESTRICTING APPARATUS describes a system in which a 
5 TV program is muted if it exceeds an allowed rating. 

US Patent No. 5,778,135 for REAL-TIME EDIT CONTROL FOR VIDEO 
PROGRAM MATERIAL describes a system in which video is segmented and each segment 
rated. An application edits out the segments that are rated above a selected level. This is 
essentially the same technique used in PCT Application WO 96/41438 for ENCODER 

1 0 APPARATUS AND DECODER APP ARAUS FOR A TELEVISION SIGNAL. 

The deletion of video content responsively to content indicators in the video 
stream is known. In prior art systems, parts of the video are simply blocked by inhibiting 
display of the video at defined intervals. Also, the applications that perform the blocking are 
predefined and implemented responsively to markers (i.e., "special codes" - Please note that 

1 5 the word "code" in the specification and prior art may refer to markers or indicators, but this 
should be distinguished based on context from the word "code" as used to refer to a 
procedure or process) in the video stream. Also, there are computer games and other kinds of 
software that play back video sequences selectively in accordance with an execution path 
resulting from a user interaction with the software. For example, there may be several 

20 alternative video sequences that can be played during the execution of a game depending on 
choices made by a user. 

There exists a need for a mechanism that allows the creator of video content 
greater control over the features relating to the selective display of video content. Current 
technology limits such control to the degree of sophistication provided at an intermediate 

25 control or delivery point. 



SUMMARY OF THE INVENTION 

Various techniques may be used to permit real time modification of a video 
stream. According to the invention, the way this modification is done increases the range of 
30 resulting modifications that are possible. It also increases the control the creator of the video 
stream has over the features provided at the receiving end. These advantages are provided by 
associating with each video stream or file one or more software procedures that are executed 
by the display or other generator such as a copying station (e.g., storing station, forwarding 
station, recording station, broadcasting station, etc.). The generator receives both the raw 
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video stream and code defining one or more procedures to be implemented such as to modify 
the video stream in some way. For example, in an embodiment, the generator is a television. 
According to this embodiment, the television receives a video signal with embedded 
procedure code. The television has an internal controller that separates the software data and 
5 the raw video data and executes the software data, which may then modify the video data. 
For example, the software data may be contained in the video blanking interval ("VBI") of an 
analog video signal or simply contained in a header file attached to the video file. In this 
embodiment the internal controller may be programmed with an Application Program 
Interface ("API") that provides a set of functions the procedure may access to create various 

10 effects. This could be a Java®-like system or an enhancement to Java®. The software data 
defines a procedure that is executed and which modifies the video data. The procedure may 
be keyed to time or segment markers in the video data to allow the procedure to identify 
portions of the video data to be modified. 

This API can provide a rich feature set or a lean feature set. It may also be 

1 5 written at a high level or a low level. For example, the API could provide a function to draw 
an object such as a flat rectangle or graded ellipse of a specified color over a specified area of 
the display only a certain time interval of the video. Such functions might take arguments 
from the procedure specifying the coordinates of the object, the size and shape, the color and 
the start and stop time segments. Another example is the application of a specified filter to a 

20 portion of the screen. The filter mask may be supplied as an argument. 

By providing either a large set of functions, or primitive functions that can be 
accessed and implemented in a large variety of different ways, the invention allows the video 
content generator to provide many features and options for distribution and use of the video 
content. One result is that the features that are available are not limited to some set that was 

25 predefined in the delivery device or output device (e.g., television) as in the prior art. Note 
that both a large set of functions of a more integrated nature or primitive functions can 
provide the same degree of flexibility. Both may be provided. 

At an abstract level, the invention provides for the association of executable 
procedures, that modify the video data, with the video itself. The association may be 

30 provided by the supply of procedures for processing the video substantially synchronously 
with the presentation of the video on the display processing equipment that ultimately 
transforms the multiplexed or compressed or coded signal into video data stream. Packaging 
the procedure code in the same or a related file may provide the association. Other 
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embodiments may create the association by supplying the code embedded in an interleave 

fashion in the video data stream whether analog or digital. 

Note that we are using the term "procedure" and "code" and other terms that 

refer to definitions of processes or potentiate processes broadly to encompass both 
5 declarative definitions and procedural definitions. Thus, we do not by any means intend to 

limit by such wording the invention to algorithms. The invention encompasses event-driven 

types of languages, object-oriented languages, etc. 

With procedures defined for each video, the range of possible modifications, 

conditions, rules, criteria, and options that are possible is obviously too large to provide a 
1 0 comprehensive list. This is a great advantage. Also, these possibilities do not have to be 

known at the time the display (recording, broadcast, etc.) device is developed. Thus, video 

content can be delivered with its own feature upgrades. Enhancements to content are enabled 

without requiring software changes in the display (or other) device. Although the number and 

types of modifications are endless. The following are a few examples for purposes of 
15 illustration. 

Video may be shipped with multiple language tracks one being selected 
according to a user profile accessed by the procedure. 

The procedure decrypts the video using profile data and a password entered by 

the user. 

20 The procedure applies a blur filter to portions of a frame during a scene of a 

movie to mask out frontal nudity. 

The procedure provides a control console that allows a user to speed up the 
display of the video according to the user's preference entered on a console generated by the 
procedure. 

25 The procedure provides a low resolution image and accepts data indicating 

payment authorization at which point it permits full resolution video to be shown. 

The procedure recognizes portions of the video signal based on pattern 

recognition, the portions containing material to be censored out, and omits those portions by 

skipping frames to make the speed of playback very fast. 
30 The procedure omits sound track segments, for example that represent 

expletives, indicated by markers in the video signal. 

A previously-unknown technique is transmitted with the video, such as a 

procedure that responds to the user profile in some special way or gives the user certain 

choices. 
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The procedure provides a text overlay on the video or Flash® animation on 
top of the video. 

The procedure retrieves commercials from a web site and displays the 
commercials at intervals during the video. 
5 The procedure further reduces the number and duration of commercials by 

providing the user a vehicle for paying down the commercials by accepting a payment for 
watching the video, similar to shareware that displays a banner ad until it is registered. 

The procedure controls reproduction rights so that the various license 
privileges that can be exercised by the user are controlled by a profile file on the machine. 
10 The common feature of all the above examples is that a program that is 

associated with the file provides the features enjoyed, rather than requiring them to be present 
or otherwise available to the display or reproduction device. The invention allows the creator 
or distributor of the video to control the display or other use of the video with great 
flexibility. 

15 In an embodiment, the procedure consists of commands that operate on 

separable portions of the video stream. The execution environment is stateless so that any 
finite number of such portions will always be copied with the procedure(s) applicable to 
it/them. Thus, no information is persisted ("persisted" is a neologism meaning "made to 
persist") across divisible portions of the media stream unless the procedure attached is 

20 capable, if it does not find that information, generating it itself. To insure that these separable 
portions are not themselves divided, the environment used to make copies of portions of the 
program must respect the indivisibility of the portions. One way to insure the indivisibility is 
not violated, is to block encode each indivisible portion so that the block cannot be read 
without every bit of it. An example of an indivisible portion could be a video segment in 

25 which a part of the video image is filtered in a certain way, for example to mask parts of a 
nude scene. A procedure might be executed to turn on application of the mask and many 
frames a procedure might be executed to turn off generation of the mask. The portion of the 
video between the turn-on and turn-off commands must not be divided lest the turn-on 
command not be activated ahead of the sensitive subject matter. 

30 An alternative way of implementing the invention is to insure that every frame 

of the video contains its own state-generating procedure code. This environment would also 
be stateless. Then, any number of frames that are copied will contain suitable code for 
applying the correct attributes to the frame. In the above example, where a filter was applied 
to a portion of each frame of a sequence, the command to apply the filter, and the filter's 
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definition, would precede each frame. In this embodiment, the environment is stateless 
between frames. This embodiment could be used with a broadcast model. In addition to 
procedures, information about the video could be encoded with the procedure data. For 
example, the title, author, description, etc. could be incorporated in it so that, any copied 
5 video sequence could contain global information about the video file from which the segment 
came. Such data need not be stored for each frame, but could be distributed over multiple 
frames. 

The invention will be described in connection with certain preferred 
embodiments, with reference to the following illustrative figures so that it may be more fully 

1 0 understood. With reference to the figures, it is stressed that the particulars shown are by way 
of example and for purposes of illustrative discussion of the preferred embodiments of the 
present invention only, and are presented in the cause of providing what is believed to be the 
most useful and readily understood description of the principles and conceptual aspects of the 
invention. In this regard, no attempt is made to show structural details of the invention in 

1 5 more detail than is necessary for a fundamental understanding of the invention, the 

description taken with the drawings making apparent to those skilled in the art how the 
several forms of the invention may be embodied in practice. 

BRIEF DESCRIPTION OF THE DRAWINGS 
20 Fig. 1 is an illustration of a user-environment in which the invention may be 

used. 

Fig. 2 illustrates an embodiment of the invention in which video data from a 
source is demultiplexed to extract data defining a procedure and then decoded and the 
procedure executed to modify the video responsively to a profile. 

25 Fig. 3 illustrates an embodiment of the invention in which video data from a 

source is demultiplexed to extract data defining a procedure, the compressed file modified by 
the procedure executed responsively to a profile, and the modified compressed file decoded. 
Fig. 4 illustrates an embodiment of the invention in which video data from a source is 
demultiplexed and data defining a procedure is obtained from independent source and in 

30 which the video file is decoded and the procedure executed to modify the video responsively 
to a profile. 

Fig. 5 is a figurative representation of a video file to illustrate features of 
certain embodiments of the invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Referring to Fig. 1, an example of a physical infrastructure capable of 
supporting the functional features of the invention includes view system 100 with a computer 
140 and various types of input and/or storage devices. The latter include a keyboard 1 12, a 
5 remote control 1 10, a removable medium such as a floppy disk, optical disk, memory card, 
etc. 120, Philips Pronto®, programmable controller, voice recognition/activated controller, 
mouse, gesture-recognition controller, etc. Data may be stored locally on a fixed disk drive 
135. Output devices may include a monitor or TV 130, speakers 131, and/or other output 
devices. The computer 140 (again any computation-capable devices as discussed in the 

10 summary of invention section) receives data 160 and/or video 170 from an external source 
which could be a broadcast transmission, a data store, the Internet, a network, a satellite 
transmission, a switched circuit transmission, or any other source of data or other signal. 
Note that by the term "computer" we mean a set top box, embedded system, storage device 
with a controller, or any digital device capable of carrying out the functional requirements 

15 discussed herein. The computer 140 executes procedures that may be stored on its data store 
135 or embedded in the data 160 and/or video 170 received from an external source or 
embedded in files transferred to the computer in the form of a data file. The procedures 
modify the video either in compressed or decompressed form. After modification, the video 
may be stored on VTR 133 or transmitted by a radio transmitter 137 as a broadcast, or 

20 displayed on the TV or monitor 130. The inputs and outputs shown are examples only. 

In another embodiment of the invention, the data 160 and video 170 can be 
transmitted by two different transmitters. Also, in a further embodiment, the data 160 can be 
distributed by multiple transmitters, whereas the corresponding video 170 is transmitted by a 
single transmitter. In this embodiment, the video is for example distributed nationally, 

25 whereas the data is distributed locally. This enables to provide different procedures with the 
video in different regions. In this embodiment, the range of the transmitter that transmitter the 
video 170 is larger than the range of the transmitters that are part of the multitude of 
transmitters that transmits the data 160. 

In the previous two embodiments, care has to be taken in synchronisation of 

30 the data 1 60 and the video 1 70, to ensure that procedures defined in the data are applied to 
the correct corresponding portions of the video stream. 

Referring now also to Fig. 2, in an illustrative embodiment, the computer 140 
receives a video file from some source which could be a cable, microwave, satellite, or other 
broadcast transmission 180, a computer such as a notebook 185, a network 190 such as the 
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Internet, a data store 195, or any other source of analog and/or digital data. These may also 
include a smart mobile phone, PDA, etc. In the current embodiment, the received data is a 
video stream. The video stream is received by a demultiplexer 205 which separates the video 
stream into an active video procedure data stream and a raw video stream. The former is 
5 applied to an active streams engine 225 and the latter to a decoder 2 1 0 (if necessary to 
decode a compressed video format). The output of the decoder 210 is applied to a process 
215 that checks a profile stored on the computer 140. The profile stores data characterizing 
the audience. If there is a match between the profile and the current video, the procedure is 
applied in a process 225 responsively to profile data to generate a modified video stream. If 

10 the profile is such as not to warrant a modification of the video stream the original 

decompressed video is output. The output stream is applied to an output device which may be 
any of a variety of different sinks. For example, the output may be a broadcast transmission 
180, a computer 185, a TV or monitor 131, or a data store 195. Output devices could also 
include a VTR as illustrated in Fig. 1 and the examples shown in Fig. 2 are merely illustrative 

15 examples. 

The demultiplexer may receive an analog or digital signal. An example of an 
analog signal is an NTSC signal from a television broadcast. In this case, a common place to 
place data is in the VBI, in which case the demultiplexer may extract the data residing in the 
VBI from the raw video stream and apply it to the active streams engine 225. The active 

20 streams engine simply runs the procedure applied to it. The active video procedure may 

consist of more code than can be packaged in a single VBI, in which case, the active streams 
engine 225 is programmed to acquire an entire procedure, the end of which may be indicated 
in a normal fashion such as by an end of file marker or other delimiter indicating that data 
preceding the delimiter represents a procedure is to be executed. Any suitable protocol may 

25 be defined to accumulate a procedure in the memory of the computer before the video 

segment to which it must be applied is reached. If the video data file is digital, the procedure 
data may be packaged as a header or interleaved in the data file or any other suitable way. If 
it is streaming data, the procedure can be sent in a header file or sent in small parcels as the 
video is buffered so that playback can begin immediately without waiting for an entire 

30 procedure or set of procedures to be loaded, the procedure(s) being accumulated over time. 
The latter accumulation scheme assumes the video to which they will be applied is not loaded 
before the procedure(s) is/are loaded. Alternatively, the procedure data can be distributed 
throughout the video file and executed by an interpreter running on the computer 140. (An 
interpreter is a program that executes instructions immediately upon receipt without 
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precompiling, for example, like the command line of a text-based operating system shell such 
as MSDOS or a command mode of a database program like dBase III.) 

Once the procedure is accumulated in memory, it may be executed 
responsively to profile data and to indicators in the video file. Referring to Fig. 5, the 
5 indicators in the video file or data stream can take various forms. Several different examples 
are illustrated in Fig. 5 which shows a file or streaming media data 501 with time advancing 
in the indicated direction. An audio sequence Audi could serve as a marker in which case a 
sound classifier could be run over the audio track until some feature is detected. Similarly, an 
image Imgi, Img2 or other signal fraction could be recognized to identify a part of the video 

1 0 stream. Even a subimage SI of a frame image 5 1 0 could classified to trigger an event. 

A marker such as Mi, M 2 , and M 3 could be written to the file. In an analog 
file, such as NTSC, the marker could be placed in the VBI. The time since the beginning of 
the data start point could be tracked and used to indicate portions of the video stream such as 
time delimiters Ti and T2. Again, a procedure 500 may be embedded in the video stream 

1 5 prior to the appearance of the part of the stream to which it is applied. For example the 
procedure 500 could be applied to the sequence demarcated by Ti and T 2 , but not to one 
indicated at M3. (Note that time is going up the page) 

Note that indicators are not necessary if the instructions are executable 
immediately upon receipt. One form of indicator is simply a place marker. In an analog 

20 stream, the marker could take the form of a watermark or an icon that is recognized in a 
portion of the video image or a datum multiplexed into the VBI. In a digital stream, the 
marker can be any suitable symbol and an indication of a temporal position. Of course in 
digital embodiments, the marker need not necessarily occupy a position in the stream that 
coincides with the application of the procedure, but it may. 

25 The decoder 2 1 0 may be a process that decompresses, decrypts, unpacks, 

unbundles or performs any other defined process that is required for access to the video data. 
The particulars of this process are not important to the practice of the invention. 

The profile may contain simply an identification of the user, information about 
the preferences of a user or the user-group (such as a household), or any of a variety of data. 

3 0 For example, the profile could indicate that the user-group is a household with a very young 
child. The active procedure could query the user before displaying highly violent or sexual 
subject matter and in the event of no response, mask or delete the potentially disagreeable 
subject matter. The profile database may include subject matter preferences that the 
procedure uses to filter a set of selectable attributes. For example, suppose the video file 
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contained many different video files all aggregated such that a particular one can be seen. 
The profile could filter these and present only one, or multiple ones, to be selected for 
viewing. 

As discussed above, in an embodiment, the procedure(s) consists) of 
5 commands that operate on separable portions of the video stream. The execution environment 
is stateless so that any finite number of such portions will always be copied with the 
procedure(s) applicable to it/them. No information is persisted between divisible portions of 
the execution environment unless the procedure attached is capable of handling it, or if the 
procedure is capable of generating it itself. In this frame-by-frame stateless embodiment 

10 described in the summary section above, the demultiplexer continuously generates 
commands as they are received. The commands are executed instantly with the 
demultiplexing or keyed to markers or inherent indicators in the video stream. 

Referring now to Fig. 3, in an alternative embodiment, the active procedure is 
applied to a compressed video stream. Decoding 310 is only done after the active procedure 

1 5 is applied to the raw video stream. In this example, the video is described as compressed, but 
it could be encrypted, bundled, or otherwise encoded. Again, although not shown, profile 
data may be supplied to the active streams engine to make the procedure responsive to data in 
the profile. In Fig. 4, the active procedure is transmitted or otherwise supplied to the active 
streams engine 425 in a parallel transmission. For example in a broadcast environment, a 

20 parallel transmission could be generated and the video modified according to it. Again, the 
synchronization could be insured by keying execution to markers or other indicia in the video 
stream. In this example, the key advantage is the short lifetime of the procedure code. The 
video is always updated according to procedures that are the most recent supplied by the 
source of the active procedure. 

25 In an environment where the statelessness requirement is not important, the 

active procedure data can be located at locations that are independent of the portions of the 
video to which they apply. One requirement, however, is that for a streaming source such as a 
TV broadcast or an Internet streaming file, the procedure must be loaded before it is needed. 
The procedure can be broken up, but all of it must be cumulated in memory before it is 

30 required. Then, the procedure code can be dumped. Preferably the code and the event that 
triggers the dumping can be encoded within the procedure itself. 

The code defining procedures is not necessarily at a high level, including 
elements that define complex predefined procedures, or at a low level, including elements 
that define small incremental procedures that must be assembled to perform useful actions. 
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The following are illustrative examples of the kinds of commands that can be executed by a 
suitable API to modify a media data stream. 

Play block bO-bl : Play a series of video blocks from block bO to block bl . 
Draw line xl , y 1 , x2, y2, W, C: Draw a superimposed line from the indicated coordinates, 
5 with the indicated weight and color. 

Draw rectangle xl, yl, x2, y2, W, C, F: Draw a superimposed rectangle from 
the indicated coordinates, with the indicated border weight and color and fill. 

Apply filter (ell, cl2, cl3, c21, c22, c23, c31, c32, c33), xl, yl, x2, y2: Apply 
a filter to a zone defined by a specified matrix over the specified region. 
1 0 Include video segment at path://filename.vid: Define an alternate stream and 

halt video to insert the alternate stream. 

It will be evident to those skilled in the art that the invention is not limited to 
the details of the foregoing illustrative embodiments, and that the present invention may be 
embodied in other specific forms without departing from the spirit or essential attributes 
15 thereof. The present embodiments are therefore to be considered in all respects as illustrative 
and not restrictive, the scope of the invention being indicated by the appended claims rather 
than by the foregoing description, and all changes which come within the meaning and range 
of equivalency of the claims are therefore intended to be embraced therein. 
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1 . A method of modifying a media information stream, comprising the steps of: 
incorporating, in said media information stream (500), a definition of a 

procedure (500) to be applied to modify a portion of said media information stream; 

implementing said procedure at a point of playback, recording, or 
5 retransmission (140, 133, 130, 137, 135) of said media information stream. 

2. A method as in claim 1 , wherein said procedure includes multiple steps and 
each step is included in said media information stream adjacent a portion of said media 
information stream to which said each step is applicable. 

10 

3 . A method as in claim 1 , wherein said step of implementing includes modifying 
a media information portion of said media information stream. 

4. A method as in claim 1 , wherein said step of implementing includes providing 
1 5 a command to an interpreter on a computer at said point of playback, recording, or 

retransmission. 

5. A method as in claim 1 , wherein said step of implementing includes inputting 
a command to an interpreter said interpreter being programmed to execute said command 

20 immediately upon said input, said interpreter being executed continuously on a computer at 
said point of playback, recording, or retransmission. 

6. A method as in claim 5, wherein an execution environment of said interpreter 
is stateless in that no variables are persisted from one command to the next. 

25 

7. A method as in claim 5, wherein an execution environment of said interpreter 
is stateless in that no variables are persisted from one set of commands preceding a video 
segment, to which said one set is applied to modify it, to a next set of commands. 
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8. A method of modifying a media information stream, comprising the steps of: 

synchronizing at least one command, which points to a procedure and resides in a data 
stream, with a media information stream; 

invoking said procedure responsively to said command at a point of playback 
5 in said media information stream such that said procedure modifies said media information 
stream; 

said synchronization being effective to insure that said procedure operates on a 
specified portion of said media information stream. 

10 9. A media information stream, comprising streaming video data with tokens 

(500) located at various points in said streaming video data, said tokens indicating commands 
such that when said media information stream is applied to an interpreter, said interpreter is 
enabled to modify said streaming video data responsively to said commands. 

15 10. A media information stream as in claim 9, wherein said interpreter is stateless 

such that each contiguous set of commands is executed beginning in the same virtual 
machine state as every other contiguous set of commands. 

11. A media information stream as in claim 9, wherein a modification of said 
20 modifying includes applying a filter to at least one frame of said streaming video data. 

12. A media information stream as in claim 9, wherein said streaming video data 
also includes markers and said commands are executable responsively to said markers. 

25 13. A media information stream as in claim 9, wherein said commands are 

responsive to features in said streaming video data. 

14. A broadcast system, comprising: 

a first transmitter (137) effective to output a media information stream; 
30 a second transmitter (137) effective to output a procedure data stream 

synchronized with said media information stream; 

said procedure data stream including data indicating at least one procedure, 
which, when executed is effective to modify said media information stream; 
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a synchronization of said media information stream and said procedure data 
stream being such as to insure that procedures defined in said procedure data stream are 
applied to specified portions of said media information stream. 

5 15. A system as in claim 1 4, wherein the second transmitter comprises a matrix of 

transmitters; each of the transmitters of the matrix having a range that is smaller than the 
range of the first transmitter. 

16. A system as in claim 1 5, wherein each of the transmitters of the matrix 
1 0 transmit different data. 

17. A system as in claim 14, wherein an initial state of a state machine 
corresponding to any series of commands in said procedure data stream and defining a 
procedure to be applied to a contiguous portion of said media information stream is a same 

15 state. 

1 8. A device for receiving and modifying a media information stream, comprising: 
a demultiplexer (205, 305, 405) with an input for receiving a combined data 

stream and for outputting a media information stream and a procedure data stream; 
20 a controller (225, 325, 425) with an input and an output; 

said controller being programmed to receive said media information stream 
and said procedure data stream and to modify said media information stream responsively to 
said procedure data stream. 

25 1 9. A method as in claim 1 8, wherein said procedure data stream includes multiple 

commands and each command is included in said combined data stream adjacent a portion 
representing a portion of said media information stream to which said each step is applicable, 
whereby said procedure data stream is synchronized with said media information stream. 

30 20. A method as in claim 1 8, wherein said controller is programmed to generate 

an interpreter process that executes commands in said procedure data stream to realize a state 
machine whose cycles are synchronized responsively to a structure of said combined data 
stream. 
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21. A method as in claim 1 8, wherein said controller is programmed to realize an 

interpreter process that is executed continuously such that commands in said procedure data 
stream are executed immediately upon being received by said controller. 

5 22. A method as in claim 2 1 , wherein an execution environment of said interpreter 

is stateless in that no variables are persisted from one command to the next. 
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23 . A method as in claim 2 1 , wherein an execution environment of said interpreter 

is stateless in that no variables are persisted from one set of commands preceding a video 
segment, to which said one set is applied to modify it, to a next set of commands. 
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